Method of developing a database of controllable objects in an environment

ABSTRACT

Disclosed herein are systems and methods for methods of developing a database of controllable objects in an environment. For example, a method includes a mobile device having a camera to capture images of objects in an environment. For each object, the method includes, in response to receiving a user selection of the object, training a machine-learning model to recognize the object. The method includes receiving a command associated with the object and receiving a plurality of images of the object and training the machine-learning model to recognize the object based on the plurality of images. The method further includes transmitting the trained model and the command to a wearable electronic device causing the wearable electronic device to save the trained machine-learning model to a data store and to associate the command with the trained machine-learning model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Application No. 63/132,001, filed Dec. 30, 2020, thedisclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

Electroencephalography is an electrophysiological monitoring method tonon-invasively record electrical activity on a human's scalp that hasbeen shown to represent the macroscopic activity of the surface layer ofthe brain underneath. A brain-computer interface (BCI) is acommunication system that can help users interact with the outsideenvironment by translating brain signals into machine commands. The useof electroencephalographic (EEG) signals has become the most commonapproach for a BCI because of their usability and reliability. However,existing BCIs tend to be uncomfortable, unsightly, and/or unwieldy towear during normal daily activity.

This document describes methods and systems that address issues such asthose discussed above, and/or other issues.

SUMMARY

The present disclosure describes embodiments related to a wearableelectroencephalography sensor and associated device control methods andmethods of developing a database of controllable objects in anenvironment. In an embodiment, a system for detecting brain waves of aperson is disclosed. The system includes a housing configured to fitover an ear of the person. The housing includes a bridge over the ear, afirst portion extending forward from the bridge to a position over atemple of the person, and a second portion extending rearward from thebridge. The system further includes a first dry electroencephalography(EEG) sensor disposed in the second portion, a second dry EEG sensordisposed in the first portion, a power supply, a processor, a camera incommunication with the processor, and a transmitter in communicationwith the processor.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, the camera isdisposed in the first portion and configured to capture images ofobjects in a field of view of the person when wearing the housing overthe ear. In some implementations, the system includes a flexibleearpiece extending downward from the housing and positioned to fit undera lobe of the ear when worn by the person and a third dry EEG sensordisposed in the flexible earpiece. The system may further include aposition sensor or an orientation sensor disposed in the housing. Insome examples, the system includes memory disposed in the housing. Thememory may hold a data store containing data representing featuresassociated with known devices, the features extracted from capturedimages of the known devices.

In an embodiment, a method of controlling one or more objects in anenvironment is disclosed. The method includes, by a processor of anelectronic device, receiving images from a camera of the electronicdevice, the electronic device having EEG sensors, and processing theimages to identify features corresponding to a known device. The methodincludes receiving brain-wave signals from at least two of the EEGsensors and comparing the brain-wave signals to measure a level of brainactivity. Upon detection of both (a) a feature corresponding to theknown device and (b) the level of brain activity deviating from abaseline by at least a threshold level, the method includes generating acommand signal configured to cause the known device to actuate andtransmitting the command signal to the known device (or a controller forthe known device).

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, processing theimages to identify features corresponding to the known device includes:extracting one or more image features from the images and comparing theimage features with one or more known features corresponding to theknown device, where the known features were extracted from previouslycaptured images of the known device. Comparing the image features withone or more known features may include determining that at least athreshold number of image features correspond with known features. Thefeature may include a symbol or pattern of symbols imprinted on asurface of the known device. The method may further include receivingthe images at a rate of at least two times per second.

In some examples, the electronic device includes an inertial measurementunit (IMU) and the method further includes, in response to determiningthat the known device has an adjustable setpoint, using the IMU toadjust the setpoint. In some examples, the electronic device includes aninertial measurement unit (IMU) and the method further includes, inresponse to determining that the known device is an electronic userinterface, using the IMU to select a location on the electronic userinterface at which the actuation will occur.

In an embodiment, a system for controlling one or more objects in anenvironment is disclosed. The system includes a wearable over-the-earelectronic device having a set of dry EEG sensors, a camera, aprocessor, and programming instructions. The programming instructionsare configured to cause the processor to receive images from the camera,process the images to identify features corresponding to a known device,and receive brain-wave signals from at least two of the EEG sensors. Thesystem compares the brain-wave signals to measure a level of brainactivity. Upon detection of both (a) a feature corresponding to theknown device and (b) the level of brain activity deviating from abaseline by at least a threshold level, the system generates a commandsignal configured to cause the known device to actuate and transmits thecommand signal to the known device (or a controller for the knowndevice).

Implementations of the disclosure may include one or more of thefollowing optional features. The brain-wave signals may include betawave signals. The feature corresponding to the known device may includea shape of a surface of the known device. The feature may include asymbol or pattern of symbols imprinted on a surface of the known device.In some examples, the system further includes an inertial measurementunit (IMU) and additional programming instructions that are configuredto cause the processor to, in response to determining that the knowndevice has an adjustable setpoint, uses the IMU to adjust the setpoint.In some examples, the system further includes an inertial measurementunit (IMU) and additional programming instructions that are configuredto cause the processor to, in response to determining that the knowndevice is an electronic user interface, uses the IMU to select alocation on the electronic user interface at which the actuation willoccur.

The wearable over-the-ear electronic device may further include ahousing configured to fit over an ear of a person, the housing includinga bridge over the ear, a first portion extending forward from the bridgeto a position over a temple of the person, and a second portionextending rearward from the bridge. The wearable over-the-ear electronicdevice may further include a first dry electroencephalography (EEG)sensor disposed in the second portion, a second dry EEG sensor disposedin the first portion, a power supply, and a transmitter in communicationwith the processor.

The wearable over-the-ear electronic device may further include a datastore containing known features corresponding to the known device, wherethe known features were extracted from previously captured images of theknown device. The programming instructions may include instructions toextract one or more image features from the received images and comparethe image features with the known features to identify the known device.The instructions to compare the image features with the known featuresmay include instructions to determine that a threshold number of imagefeatures correspond with known features.

In an embodiment, a method of developing a database of objects in anenvironment is disclosed. The method includes using a mobile devicehaving a camera to capture images of objects in the environment. Foreach object, the method includes receiving, via a user interface, a userselection of the object and, in response to receiving the userselection, creating a pattern of recognizable features of the object.The pattern of recognizable features is created by identifying featuresrelated to the object in one or more of the images and transmitting thefeatures to a wearable electronic device via a communication linkbetween the mobile device and the wearable electronic device, causingthe wearable electronic device to save the pattern to a data store inthe wearable electronic device.

Implementations of the disclosure may include one or more of thefollowing optional features. In some examples, the method furtherincludes receiving an executable command associated with the selectedobject and transmitting the executable command to the wearableelectronic device via the communication link, which causes the wearableelectronic device to associate the executable command with the pattern.The executable command, when executed by the wearable electronic device,may cause the wearable electronic device to adjust a setting of theobject. Identifying the features may include locating one or morekeypoints in the one or more images and assigning identifiers to thekeypoints. The identifiers may include Binary Robust IndependentElementary Features (BRIEF) descriptors. Receiving the user selection ofthe object may include receiving a user selection of an image of theobject.

In an embodiment, a method of training a machine-learning model isdisclosed. The method includes using a mobile device having a camera tocapture images of objects in an environment. For each object, the methodincludes receiving, via a user interface, a user selection of theobject, receiving a command associated with the selected object and, inresponse to receiving the user selection, training the machine-learningmodel to recognize the object. The method trains the machine-learningmodel to recognize the object by receiving a plurality of images of theobject and training the machine-learning model based on the plurality ofimages. The method further includes transmitting the trained model andthe command to a wearable electronic device via a communication linkbetween the mobile device and the wearable electronic device, causingthe wearable electronic device to save the trained machine-learningmodel to a data store in the wearable electronic device and to associatethe command with the trained machine-learning model.

Implementations of the disclosure may include one or more of thefollowing optional features. Causing the wearable electronic device toassociate the command with the trained machine-learning model mayfurther cause the wearable electronic device to execute the command inresponse to the trained machine-learning model recognizing the object inan image captured by the wearable electronic device. The command, whenexecuted, may cause the object to change from a first state to adifferent state. The machine-learning model may include a neuralnetwork. Receiving the plurality of images may include capturing a videoof the object at multiple angles. The machine-learning model may beconfigured to determine a probability that the object is in the image,and the machine-learning model may be further configured to recognizethe object when the determined probability satisfies a probabilitythreshold.

The method may further include determining one or more regions ofinterest of one or more of the plurality of images and training themachine-learning model based on the portion of the plurality of imageswithin the one or more regions of interest. In some examples, the methodincludes receiving a reference image of the selected object andidentifying reference features related to the selected object in thereference image. The method may further include constructing a boundingbox around the object in one or more of the plurality of images,extracting features from the portion of each of the images within thebounding boxes, and comparing the extracted features with the referencefeatures. In response to a threshold number of extracted featuresmatching reference features, the method may include training themachine-learning model on the portion of the image within the boundingbox. In some examples, the method includes adjusting a dimension,position, or orientation of the bounding box around the object until thethreshold number of extracted features match reference features.

In an embodiment, a system for training a machine-learning model isdisclosed. The system includes a mobile device having a camera tocapture images of objects in an environment. The mobile device isconfigured to receive, via a user interface, a user selection of anobject, receive a command associated with the selected object, and trainthe machine-learning model to recognize the object. The system trainsthe machine-learning model to recognize the object by receiving aplurality of images of the object, training the machine-learning modelbased on the plurality of images and transmitting the trained model andthe command to a wearable electronic device via a communication linkbetween the mobile device and the wearable electronic device, causingthe wearable electronic device to save the trained machine-learningmodel to a data store in the wearable electronic device and to associatethe command with the trained machine-learning model.

Implementations of the disclosure may include one or more of thefollowing optional features. Receiving the plurality of images mayinclude capturing a video of the object at multiple angles. The mobiledevice may be further configured to determine one or more regions ofinterest of one or more of the plurality of images and train themachine-learning model based on the portion of the plurality of imageswithin the one or more regions of interest. In some examples, the mobiledevice is configured to receive a reference image of the selected objectand identify reference features related to the selected object in thereference image. The mobile device may construct a bounding box aroundthe object in one or more of the plurality of images, extract featuresfrom the portion of each of the images within the bounding boxes andcompare the extracted features with the reference features. In responseto a threshold number of extracted features matching reference features,the mobile device may train the machine-learning model on the portion ofthe image within the bounding box. The mobile device may be configuredto adjust a dimension, position, or orientation of the bounding boxaround the object until the threshold number of extracted features matchreference features.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example environment for controlling a device based onbrain activity;

FIG. 2 shows an example wearable EEG apparatus;

FIG. 3 is a flowchart of a method of detecting brain activity;

FIG. 4 is a flowchart of a method of detecting devices to control;

FIG. 5 is a flowchart of a method of training a machine-learning model;

FIGS. 6A and 6B show an example environment for controlling a cursorposition based on brain activity;

FIG. 7 shows an example environment for managing a data store of theexample wearable EEG apparatus;

FIG. 8 shows details of the data store;

FIGS. 9A and 9B show an example environment for controlling a devicebased on brain activity; and

FIG. 10 illustrates a block diagram of internal hardware included in anyof the electronic components of this disclosure.

DETAILED DESCRIPTION

As used in this document, the singular forms “a,” “an,” and “the”include plural references unless the context clearly dictates otherwise.Unless defined otherwise, all technical and scientific terms used hereinhave the same meanings as commonly understood by one of ordinary skillin the art. As used in this document, the term “comprising” (or“comprises”) means “including (or includes), but not limited to.” Whenused in this document, the term “exemplary” is intended to mean “by wayof example” and is not intended to indicate that a particular exemplaryitem is preferred or required.

In this document, when terms such “first” and “second” are used tomodify a noun or phrase, such use is simply intended to distinguish oneitem from another and is not intended to require a sequential orderunless specifically stated. The term “about” when used in connectionwith a numeric value, is intended to include values that are close to,but not exactly, the number. For example, in some embodiments, the term“about” may include values that are within +/−10 percent of the value.

The present disclosure relates generally to methods and systems fordeveloping a database of controllable objects in an environment,detecting brain activity, and controlling one or more electrical orelectronic devices based on the detected brain activity. FIG. 1 shows anexample environment 100 for controlling a device 130 based on brainactivity. The environment 100 includes a user 102 associated with awearable EEG apparatus 200. Here, the wearable EEG apparatus 200 isaffixed to an ear of the user 102 and is configured to monitor brainactivity of the user 102 via one or more EEG sensors 220 (FIG. 2). Thewearable EEG apparatus 200 is in wireless communication with a devicecontroller 120 capable of controlling one or more functions orcharacteristics of a device 130. The wireless communication may followcommunication protocols such as Near Field Communication (NFC),Bluetooth, WiFi, Infrared (IrDA), or other technology allowing fortransmitting and/or receiving information wirelessly. The devicecontroller 120 may be capable of receiving instructions and, in responseto receiving the instructions, controlling, operating, or adjusting thedevice 130 according to the instructions. Here, the device controller120 is capable of turning a light bulb on and off. In other examples,the device controller 120 may adjust a setpoint, such as the temperaturesetting of a climate control system, or the station and/or volume of atelevision or audio system. The device controller 120 may adjust morecomplex parameters, such as a time profile of a setting. The devicecontroller 120 may also adjust a virtual setting, such as the positionof a cursor or other object on a computer screen, or the remaining timeon a countdown timer. The device controller 120 includes a switch, atransistor, or another device (such as a microprocessor or integratedcircuit) that is capable of causing the device 130 to change acondition. For example, the device controller 120 may be capable ofclosing a circuit that includes the device 130 and a power source, andalso opening the circuit between the device 130 and power source, thusturning the device on and off. The device controller 120 also mayinclude a circuit that can adjust one or more functions of the device130, such as a light brightness control, a volume control, a fan speedcontrol, or other control device. Other examples of devices 130 that maybe controlled by the device controller 120 include remote controlleddoors, windows, skylights, blinds or shades, etc., kitchen appliancessuch as toasters, blenders, ovens, range tops, garbage disposals, trashcompactors, plumbing devices such as sinks, showers, or toilets,Internet of Things (IoT) devices such as smart plugs, and any otherdevice that can be controlled using a computer.

In some embodiments, the wearable EEG apparatus 200 is in wirelesscommunication with a computing device 104 associated with the user 102.The computing device 104 may be capable of directly controlling one ormore functions or characteristics of the device 130. The computingdevice 104 may transmit instructions to the device using a wired orwireless communication channel. For example, the computing device 104may transmit an infrared or RF signal to control audio, video, or otherelectronic equipment, e.g., to cause the device to change its state. Forexample, if the device is a television, the command may be to actuatethe television's power switch and thus from on to off) or vice versa, toor change to a different station. The computing device 104 may also sendan audio signal to a voice-controlled device, such as a virtualassistant. Alternatively, the computing device 104 may be incommunication with the device controller 120, allowing the computingdevice 104 to control one or more functions or characteristics of thedevice 130 via the device controller 120. For example, the computingdevice 104 may transmit a signal (such as an optical signal or acommunication with instructions) to turn a smart outlet on or off orcontrol a smart appliance, such as an oven, crockpot, clotheswasher/dryer, refrigerator, garage door opener, dishwasher, vacuumcleaner, alarm system, climate control system, or the like. Examples ofcomputing devices include smartphones, tablets, laptop computers, orother devices capable of wirelessly communicating with the wearable EEGapparatus 200 and directly or indirectly controlling one or morefunctions or characteristics of the device 130, e.g. by interfacing withthe device controller 120.

The wearable EEG apparatus 200 also includes an imaging system 240 (FIG.2), such as a camera, capable of receiving images of objects in thefield of view 114 of the imaging system 240. The imaging system 240 maybe configured to capture images of objects in the field of view of theuser. Here, the light bulb is with the field of view 114 of the imagingsystem 240. The wearable EEG apparatus 200 may detect brain activity ofthe user 102 indicating that the user 102 is focusing on an object inthe field of view of the user (e.g., the light bulb). In response todetecting the brain activity, the wearable EEG apparatus 200 maytransmit a signal to the device controller 120, causing the devicecontroller 120 to turn the light on. The signal may be a digital signalthat transfers programming instructions and/or an encoded data packetthat, when received by a processor of the device controller 120, willcause the processor to trigger the device controller 120 to take anaction, such as turn the device 130 on or off. The signal also may be atimed sequence of pulses (such as those of a modulated infrared signal)that, when received by the device controller 120 and recognized by itsprocessor, will cause the processor to activate or deactivate or changethe state of the device 130.

FIG. 2 shows an example wearable EEG apparatus 200. The apparatus 200includes a housing 210 supporting and/or containing other components ofthe apparatus 200. The housing 210 may be configured to be worn over theear of a user 102. The housing 210 may be formed from a rigid orsemi-rigid material, such as a plastic, and may be curved or otherwiseshaped to conform with the side of the user's head. The housing 210 maybe an elongated structure with a forward section 211, a central section212 and a rear section 213. When the device is worn over a user's ear,forward section 211 will extend from the user's ear toward the user'sface, and rear section 213 will extend from the user's ear toward theback of the user's head and/or neck. Central section 212 serves as abridge over the user's ear by joining the forward section 211 and therear section 213 and including a semicircular indentation 230 that issized and positioned to be placed over the ear, behind (and optionallyextending upward from) the helix of the ear. Optionally, thesemicircular indentation 230 may include a liner 235 formed of a gel, anelastomer such as silicone or other elastic or rubber-like material, orother resilient material to provide a more comfortable fit, and also tohelp keep the device from falling off of the user's ear during use.

The housing 210 may include mounting positions for two or more EEGsensors and an imaging system 240. Here, three EEG sensors 220 a, 220 b,220 c that are dry EEG sensors are mounted on the housing 210. (Thisdocument may refer to the EEG sensors collectively using referencenumber 220.) Dry EEG sensors are capable of operating without the use ofconductive gel or paste. However, EEG sensors that are wet EEG sensorsare also within the scope of the disclosure. In some examples, wearableEEG apparatus 200 is configured to locate the three EEG sensors 220 a,220 b, 220 c near locations T4, F8, and A2, respectively, ofInternational 10-20 system of describing the location of scalpelectrodes.

As shown, a first EEG sensor 220 a is mounted on the central section 212of the housing 210 so that, when the wearable EEG apparatus 200 is wornby the user 102, the first EEG sensor 220 a will contact the user 102 ata position behind the user's ear (i.e., a location under the ear's helixor between the helix and the back on the user's head), over a temporallobe of the user's brain.

A second EEG sensor 220 b is mounted on the forward section 211 of thehousing 210 so that the second EEG sensor 220 b will contact the user102 at a position at or near the user's temple, over a frontal lobe ofthe user's brain.

A third EEG sensor 220 c is mounted on the housing 210 so that the thirdEEG sensor 220 c will contact the user 102 at a position that isrelatively lower on the user's head than the positions of the first EEGsensor 220 a and second EEG sensor 220 b below the first EEG sensor 220a. For example, the position of the third EEG sensor 220 c maycorrespond to a location that is under the user's earlobe when worn. Insome examples, the third EEG sensor 220 c is mounted on a flexibleearpiece 250 extending away (e.g., downward) from either the centralsection 212 or the rear section 213 of the housing 210. The flexibleearpiece 250 may be formed from a gel, an elastomer such as silicone orother elastic or rubber-like material, or other materials, and it may beintegral with or separate from liner 235. The flexible earpiece 250 mayincluding wiring leading from the electrical components within thehousing 210 to the third EEG sensor 220 c. In some examples, the wiringincludes Benecreat aluminum wiring. The flexible earpiece 250 may becovered with a cloth and/or padding to provide additional comfort. Theflexible earpiece 250 may bend to facilitate placing the wearable EEGapparatus 200 over an ear of the user 102 and securely attaching thewearable EEG apparatus 200 to the user 102. In some examples, theflexible earpiece 250 is capable of securing the wearable EEG apparatus200 to the user 102 even during vigorous activity. The flexible earpiece250 may apply a gripping force to the ear of the user 102 when the user102 is wearing the wearable EEG apparatus 200. In some examples, thehousing 210 is configured to translate the gripping force of theflexible earpiece 250 to the EEG sensors 220, causing the EEG sensors220 to press firmly against the user 102 to facilitate receiving astrong EEG signal.

As shown in FIGS. 1 and 2 together, an imaging system 240 is mounted atthe front of the front section 211 of the housing 210, with a lens thatis positioned toward the user's face and configured to capture images ofobjects in the field of view 114 of the imaging system 240. The imagingsystem 240 may be mounted on the housing 210 so that the field of view114 includes devices 130 that are within the field of view of the eyesof the user 102 when the user 102 is wearing the wearable EEG apparatus200.

The housing 210 may enclose other components of the wearable EEGapparatus 200, e.g., to protect the components from damage or exposureto hazards. The components may include items such as those illustratedin FIG. 10, including a communication device 1010 that includes atransmitter, a processor 1005 in communication with the transmitter, theEEG sensors (220 in FIG. 2, 1020 in FIG. 10), and the imaging system(240 in FIG. 2, 1040 in FIG. 10). The components may also include wiringand interfacing components, such as signal conditioning components(e.g., band-pass filters), some of which may be mounted on printedcircuit boards (PCBs) enclosed within the housing 210. The processor1005 may have associated memory containing programming instructionsthat, when executed, cause the processor 1005 to implement methods, suchas detecting brain activity and controlling one or more devices 130based on the detected brain activity. The components may also include apower source 1080 (e.g., a rechargeable battery) providing power toelectrical components of the wearable EEG apparatus 200. The battery maybe sized to allow for extended use of the wearable EEG apparatus 200between charges. Furthermore, the processor 1005 may be a low-powercomponent, e.g., designed for use in a mobile phone. The battery may becharged though an electrical connection or by inductive wirelesscharging, and the wearable EEG apparatus 200 may harvest energy from theenvironment to extend the time between recharges. In some examples, thewearable EEG apparatus 200 includes a photovoltaic device disposed on aportion of the housing 210.

As noted above, in some examples, three EEG sensors 220 are disposed onthe housing 210 so that the EEG sensors 220 contact the user 102 atspecific locations on the user's head. For example, the third EEG sensor220 c may contact the user 102 below the user's ear, at a location oflow EEG signal strength. At this location, the third EEG sensor 220 cmay act as a ground reference relative to the other two EEG sensors 220a, 220 b by being placed on or near the midline sagittal plane of theskull, where lateral hemispheric cortical activity is largely notdetected. The second EEG sensor 220 b may contact the user near theuser's temple, at a location of relatively high EEG signal strength. Insome examples, a filter, such as a band-pass filter, attenuates lessrelevant signals from the second sensor 220 b. For example, theband-pass filter may pass beta waves (e.g., signals in the range from20-30 Hz), while attenuating delta, theta, gamma, and alpha waves. Insome examples, the processor 1005 may adjust parameters of the filter,e.g., to pass alpha wave signals in lieu of beta wave signals or toswitch between passing alpha wave signals and passing beta wave signals.The first EEG sensor 220 a may contact the user behind the user's ear ata location of relatively lower EEG signal strength than the location ofthe second EEG sensor 220 b.

Some variations of the devices may have additional EEG sensors locatedin different locations. Other devices may have fewer than three sensors.For example, the system may operate with only the first EEG sensor 220 apositioned over the user's temple and frontal lobe and the third EEGsensor 220 c providing a ground reference. Alternatively, the system mayoperate with only the second EEG sensor 220 b positioned under the helixof the user's ear and over the user's temporal lobe and the third EEGsensor 220 c providing a ground reference.

FIG. 3 shows a method 300 of detecting brain activity to determine whenthe brain activity of a user of the device indicates an increase in theuser's focus. At step 302, the method 300 includes receiving, by theprocessor of the wearable EEG device or of a separate electronic devicethat is in communication with and proximate to the wearable EEG device,the filtered EEG signals from one or more EEG sensors 220. The processormay sample the filtered brain-wave signals, e.g., using ananalog-to-digital converter (ADC) to determine a representativeamplitude 304 of the brain wave signal in the pass band. Therepresentative amplitude may be a root mean square (rms) value or othervalue representing the level of brain activity within the frequencyrange of the pass band. In some examples, as time progresses theprocessor determines a difference 306 between the representativeamplitude of the brain wave signal from the first EEG sensor 220 a andthe second EEG sensor 220 b to determine the level of brain activity.The filter and ADC may be disposed on a daughter PCB interfaced with theprocessor PCB. In some examples, the EEG sensors filter and sample thebrain wave signals and provide digital representations of the signals tothe processor, e.g., through a serial or parallel data bus. Theprocessor may receive 302 EEG signals at irregular or regular intervalsof time, such as twice per second. At step 306, the processor mayfurther measure a change in the representative level of brain activitybetween interval by comparing one level of a first interval with asecond level of a previous (or subsequent) interval.

In some examples, the processor compares the measured change inrepresentative level of brain activity to a threshold 308. When thechange in representative level of brain activity exceeds the threshold,the processor may determine that the user's focus has increased. Thethreshold may be based on a baseline level of brain activity of theuser. In some examples, the user triggers the wearable EEG apparatus 200to determine the baseline level of brain activity of the user, e.g.,through an application 710 (FIG. 7) executing on the computing device orthrough an actuator on the wearable EEG apparatus 200, e.g., by pressinga button located on the housing 210. In other examples, wearable EEGapparatus 200 automatically determines the baseline level of brainactivity of the user 102, e.g., as an average or mean level over aperiod of time, with fluctuation in the representative level of brainactivity of the user 102 over that period of time.

In some examples, wearable EEG apparatus 200 includes a data store 800(FIG. 8) containing descriptive information for known devices. Referringback to FIG. 2, the imaging system 240 is configured to capture imagesof objects in the field of view 114 of the imaging system 240. When adevice 130 is in the field of view of the imaging system 240, thewearable EEG apparatus 200 may recognize the device 130, based on imagescaptured by the imaging system 240, and select the device 130 from theknown devices contained in the data store 800. Example image recognitionmethods will be described below in the discussion of FIG. 4.

At step 310, in response to the user's increase in focus, the processor1005 may perform functions to control the selected device 130. In someexamples, the data store 800 also contains patterns or features 808(FIG. 8) associated with each known device. The pattern may be apreviously captured image of the device 130, or the pattern may includefeatures of the device 130 extracted from the previously captured imageof the device 130, e.g., by image processing. The extracted features mayinclude colors of surfaces of the device 130, geometric features, suchas detected edges of the device 130, or patterns, textures, or shapes ofa surface of the device 130. In some examples, the pattern includes anApril tag, barcode, QR code, or other form of computer-readable dataassociated with the device 130, e.g., affixed to a surface of thedevice. In some examples, the extracted feature includes a symbol orpattern of symbols imprinted on the surface of the device 130, such as alogo. The extracted features may include knobs, switches or othercontrols associated with the device 130 or text or other displayinformation associated with the device 130.

FIG. 4 shows a method 400 of detecting devices 130 to control. At step402, the processor 1005 may receive images from the imaging system 240and may extract, at step 404, features from the received images.Features may include characteristics of the image, such as overall shapeas well as corners, edges, relationships between corners and edges,colors, patterns, etc. In some examples the processor 1005 executes aScale Invariant Feature Transform (SIFT) or similar algorithm to locateone or more “keypoints” of the image. A “keypoint” is a feature which isunaffected by the size, aspect ratio or other asymmetry of a shape inthe image. The processor may construct a scale space to ensure that thekeypoints are scale-independent, ensure the keypoints are rotationinvariant (unaffected by the orientation of the image), and assign aunique identifier to the keypoint. In some embodiments, the uniqueidentifier may be a 128 bit or 256 bit Binary Robust IndependentElementary Features (BRIEF) descriptor. In some examples, the processorexecutes a Features from Accelerated Segment Test (FAST)corner-detection algorithm to extract features. The processor 1005 mayexecute functions of a software library, such as the python module ORB(Oriented FAST and Rotated BRIEF), to perform feature extraction. Theprocessor 1005 may compare, at step 406, the received image and/or thefeatures extracted from the received image with features associated withknown devices contained in the data store 800. For example, theprocessor 1005 may compare unique identifiers assigned to keypoints ofthe image with unique identifiers assigned to keypoints of previouslycaptured images of known devices. The processor 1005 may select, at step408, a device 130 from the known devices when a threshold number ofextracted features match features of previously captured images. Forexample, each known image contained in the data store 800 may include,e.g., 30 or more features 808 extracted from a previously captured imageof the device 130. In this example, the processor 1005 may select thedevice 130 from the known devices when more than ten features match. Theprocessor 1005 may also select the device 130 having the greatest numberof matching features when more than one of the known devices has morethan the threshold number of matching features extracted from thecaptured image of the device 130.

Alternatively, to classify the device at 408 the wearable EEG apparatus200 may use a machine-learning algorithm to recognize the device 130.For example, the processor 1005 may provide the received image to adeep-learning neural network, such as a convolutional neural network(CNN) trained to recognize known objects. FIG. 5 shows a method 500 oftraining a machine-learning model (such as a deep-learning neuralnetwork) to recognize a known device 130 to control. In some examples,the user (or other supervised trainer) trains the deep-learning neuralnetwork using several images of the known object. At step 502, themethod 500 includes receiving an object selection. For example, the usermay cause the system to select an object in the scene as the knowndevice 130 by acquiring an image of the object, e.g., through anapplication 710 (FIG. 7). The user may label or annotate the selectedobject via the application 710. The system may first acquire a singlereference image of the object, and extract features from the image asdescribed above. At step 504, the method 500 includes receiving objectimages. For example, the user may cause the system to acquire one ormore still images or to acquire a continuous video at a fixed framerate, such at 60 frames per second, for a period of time, such as 10seconds). The system may acquire the additional images at various angleswith respect to the object, at various distances from the object, undervarious lighting conditions, and/or other circumstances providingvariation of the appearance of the object in the image. For example, thesystem may acquire a video of the object while the user walks around theobject and/or walks toward or away from the object, and/or whileincreasing or decreasing the light shining on the object. In someexamples, the system receives the additional images from a data store ofimages of the object. These additional images serve as additionaltraining images for training the deep-learning network to recognize theobject as the known device 130.

In some examples, the system then performs image processing on theadditional images to prepare the additional images for use as trainingimages. For example, the system may determine a region of interestaround the known device and crop out portions of additional images whichare unrelated to the known device. In some examples, the user may drawbounding boxes around the known device in the additional images, and inresponse the system disregards the portion of the images outside thebounding boxes. In some examples, bounding boxes are determinedautomatically using image processing. For example, the system may applyan image-processing algorithm to extract features, such as keypoints, ofobjects in the first (reference) image. The system may then compare thefeatures extracted from the reference image to features extracted fromthe additional images. The system may apply one or more bounding boxesto the additional images and adjust aspects of the bounding boxes, suchas dimensions and orientation until a threshold number of featuresextracted from the portion of the additional images within the boundingbox match features extracted from the reference image. For example, thesystem may scale dimensions of the bounding box up and down, or theposition or orientation of the bounding box up and down, back and forth,and/or through a range of linear or angular values while processing theimage. The system may discard ranges of dimensions and/or angles of thebounding box that are not associated with the threshold number ofmatching features. One set of dimensions and/or orientations associatedwith the threshold number of matching features may be retained fortraining. If no bounding box results in the threshold number of matchingfeatures (i.e., if the threshold number of matching features cannot beextracted from the entire image), the system may discard the referenceimage itself. Thus, the process that the system follows to implement theimage-processing algorithm disregards images or portions of theadditional images unrelated to the known device prior to training thedeep-learning network. At step 506, the method 500 includes training amachine-learning model. The portion of the additional images containedwithin the bounding boxes may be input into the deep-learning neuralnetwork, thus training the deep-learning neural network to recognize theknown device in a variety of circumstances (e.g., including variousmagnifications, orientations, and lighting). After training, thedeep-learning neural network may predict the probability that the knowndevice is in a newly acquired image. At step 508, the method 500includes transmitting the trained deep-learning neural network to thewearable EEG apparatus 200. One or more trained deep-learning neuralnetwork may be stored in the data store 800 of the wearable EEGapparatus 200 and applied to images acquired by the wearable EEGapparatus 200. In some examples, one or more deep-learning neuralnetworks may determine a probability that the known device 130 is in thenewly acquired image. The one or more deep-learning neural networks maydetect that the known device 130 is in the newly acquired image when thepredicted probability satisfies or exceeds a threshold. At step 510 and512, the method 500 includes receiving a command associated with theknown device 130 and transmitting the associated command to the wearableEEG apparatus 200. The user may provide commands or functions associatedwith one or more deep-learning neural network, as described in FIG. 8,so that when the deep-learning neural network recognizes an object, thewearable EEG apparatus 200 executes the associated command or function.

When the wearable EEG apparatus 200 classifies an object and the objectsatisfies one or more other threshold conditions, at 410 the system mayselect the device as an object that the EEG apparatus 200 will control.For example, the system may detect multiple devices in a field of viewof the EEG apparatus's camera. To determine which of the devices will becontrolled, the system may require that the device to be controlled iscentrally positioned within the field of view, such as over the centerpoint or within a threshold distance of the center point of the field ofview, for at least a threshold period of time. The system also mayrequire that the user's focus have increased while the device is withinthe central location or the field of view in order to select the deviceto control. (Methods of determining whether focus has increased aredescribed above in the discussion of FIG. 3). Once the EEG apparatus 220selects the device 130 as a device to be controlled, the wearable EEGapparatus 200 may control 412 one or more functions or characteristicsof the device 130.

The wearable EEG apparatus 200 may further include an accelerometer,Inertial Measurement Unit (IMU), or other position/orientation sensor260 (FIG. 2) in communication with the processor 1005 and providingposition/orientation information to the processor 1005. In someexamples, the IMU is disposed within the housing 210 at a positionbetween the user's ear and the user's temple, or other suitable locationwhich tracks well with movement of the user's head. The processor 1005may determine that the device 130 includes an electronic interfacecapable of receiving position/orientation information. For example, theprocessor 1005 may determine that device 130 is capable of discovering aBluetooth mouse, trackball, or other wireless position-controllingcomputer peripheral. The processor 1005 may cause the wearable EEGapparatus 200 to interface with (e.g., pair with) with the device 130and use information from the position/orientation sensor 260 to controla cursor location or other point of focus associated with the device130. For example, the wearable EEG apparatus 200 may transmit relativeX-axis and Y-axis data to the device 130, e.g., according to theBluetooth Human Interface Device (HID) profile. The X-axis and Y-axisdata may indicate a number of pixels to move the device's cursor fromits current position. The user 102 of the EEG apparatus 200 may adjustthe cursor location on the display screen by changing the orientation ofthe wearable EEG apparatus 200, thus changing the orientation of theposition/orientation sensor 260. For example, tilting the user's head upand down will cause a corresponding change in orientation of the IMU.The system will measure changes in one or more values of theposition/orientation as the user's head tilts up and down. The systemmay scale the measured changes of position/orientation by aproportionality constant to determine a number of pixels to move thecursor up or down. The system then sends the scaled measurement data as,e.g., relative Y-axis data to the device 130, causing the cursor to moveup and down in response to the user's head tilting up and down.Similarly, the system may send relative X-axis data to the device 130 asthe user's head moves side to side, causing the device's cursor to moveback and forth.

For example, FIG. 6A shows an example environment 600 for controllingthe location of a cursor 604 on a controllable display screen 602 of acomputer system. The environment 600 includes a controllable displayscreen 602 of a computer system displaying a selectable window 606 and acursor 604. The computer system may include a device controllerapplication and communicate directly with the EEG apparatus vianear-field or other wireless communication protocols. Alternatively, theenvironment 600 also may include a portable electronic device 614 onwhich a device controller application is installed. Once the EEGapparatus 200 recognizes the display screen 602 using processes such asthose described above in the discussion of FIG. 4, it may then usemovement of the EEG apparatus 200 to control movement and activation ofthe cursor 604. The cursor 604 is shown as located near the lower rightcorner of the display screen 602 in FIG. 6A, but in practice the cursor604 may start in any position of the display screen 602.

The user 102 of the EEG apparatus 200 may adjust the cursor location onthe display screen by changing the orientation of the wearable EEGapparatus 200, thus changing the orientation of the position/orientationsensor 260. The device controller application of the portable electronicdevice 614 or of the computer system with then generate a command tomove the cursor to a location that corresponds to movement of theposition/orientation sensor. For example, referring to FIG. 6B, thecursor 604 has been moved to a higher location on the computer screen602 in response to the wearable EEG apparatus 200 tilting upward. Insome examples, the system will determine a reference point, such as theposition of the cursor 604 on the display screen 602 when the systemdetects a certain trigger, such as a determination that the user's focushas remained at or above a threshold for at least a minimum amount oftime, or a determination that the cursor 604 and/or the display screen602 is centrally located (i.e. within a threshold range of a centerpoint) in a field of view of the imaging system of EEG apparatus 200 forat least a minimum period of time. When the trigger happens, the systemwill determine initial position and/or orientation values of the EEGapparatus' position/orientation sensor and save that information toonboard memory. The system will measure changes in one or more values ofthe position/orientation sensor as compared to the reference point.

The processor may further determine that device 130 is capable ofreceiving a select event (e.g., mouse left click), e.g., according tothe Bluetooth Human Interface Device (HID) profile. The processor maycause the wearable EEG apparatus 200 to transmit a select event to thedevice 130 in response to determining that the user's focus hasincreased. In the example shown, the user may position the cursor 604within the window 606 (e.g., over the selectable close button, shown asan X in the title bar of the window) and select the window or a functionof the window (e.g., selecting the close function to close the window).In this way, wearable EEG apparatus 200 may control large presentationscreens, video games, mobile computing platforms such as smart phones,smart televisions, automobile infotainment systems, and other electronicdevices with movable cursors. The user 102 may position the cursor at adesired location, then, by focusing on the location, cause the positionor object on the screen to be selected. In some examples, the wearableEEG apparatus 200 will pair with multiple devices 130 simultaneously. Insome examples, the wearable EEG apparatus 200 will only pair with thedevice 130 if the device 130 is contained in the data store 800 of thewearable EEG apparatus 200 and is recognized by the wearable EEGapparatus 200.

FIG. 7 shows an example environment 700 for managing the data store 800of the wearable EEG apparatus 200. Here, a computing device 104associated with the user 102 may execute an application 710 whichmanages the known devices in the data store 800 of the wearable EEGapparatus 200. In some examples, the application 710 executes on aremote server, such as a web server that the user 102 accesses via thecomputing device 104. The application 710 may capture images of objects,such as the device 130, in the field of view 714 of an imaging system ofthe computing device 104 and extract features from the captured image ofthe device 130, e.g., by image processing. In some examples, images ofdevices 130 are uploaded to the computing device 104 through a computernetwork or bus. The application 710 may present images of devices 130 tothe user 102. In response, the user 102 may select the device 130 as aknown device to include in the data store 800 of the wearable EEGapparatus 200. In response, the application 710 may cause the computingdevice 104 to transmit the selected device 130 (and patterns associatedwith the selected device 130) to the wearable EEG apparatus 200. Thewearable EEG apparatus 200 may add the selected device and patterns 808associated with the selected device, such as the captured image and/orthe extracted features, to the data store 800. In some examples, thecomputing device 104 receives a list of known devices contained in thedata store 800 of the wearable EEG apparatus 200. The application 710may present the list to the user 102, so that the user 102 may selectdevices to be removed from the list of known devices contained in thedata store 800. In response to selections by the user 102, theapplication 710 may cause the computing device 104 to transmit theselected devices to the wearable EEG apparatus 200, causing the wearableEEG apparatus 200 to remove the selected devices from the data store800. Using the application 710 in this way allows the user to add,remove, and update objects included in the list of known devicescontained in the data store 800 of the wearable EEG apparatus 200.

FIG. 8 shows details of the data store 800. Each known device 130 mayhave an associated device ID 802. In some examples, the list of knowndevices includes functions 804 associated with each device 130. Examplefunctions include activate/deactivate, open/close, step to a nextelement or position in a sequence (e.g., tune a radio or televisionreceiver to a next station, or toggle through setting of COOL, OFF, andHEAT of a climate control system, or speeds of a cooling fan). Somedevices 130 may have more than one associated function 804. For example,a cooling fan having multiple fan speeds may also include a light whichcan be turned on and off. As shown in FIG. 7, the light bulb can betoggled between the ON state and the OFF state. In addition, a motionsensor associated with the light bulb (which may activate the light whenit detects motion near the light) can also be toggled between an activeand inactive state. In some examples, the application 710 allows theuser 102 to select the function performed by the wearable EEG apparatus200 in response to the user's increase in focus. Here, the function thestate of the light bulb is selected. The application 710 may provide alist of functions associated with each known device for the user toselect from. The application 710 may allow the user to define additionalfunctions 804 associated with each known device (or group of knowndevices). For example, the application 710 may allow the user to enteradditional functions according to the syntax of a command language orscript language, such as TCL, perl, or python, or according to thesyntax of a general-purpose computing language, such as Java. Theapplication 710 may cause the computing device 104 to transmit thedefined and/or selected functions to the wearable EEG apparatus 200,causing the wearable EEG apparatus 200 to execute the function inresponse to the user's increase in focus. In some examples, theapplication 710 may download functions from a repository of functionsassociated with a device or class of devices. The application 710 mayalso upload functions to the repository, e.g., after a suitable degreeof testing.

In some examples, the list of known devices includes settings 806associated with each device 130. The processor 1005 may determining achange in position or orientation of the wearable EEG apparatus 200,e.g., based on information received from a position/orientation sensor260. In response to determining a change in position or orientation, theprocessor 1005 may perform functions to control a setting of theselected device 130. For example, in response to the user's head movingfrom a downward facing position to a more upward facing position, theprocessor 1005 may cause a volume setting of the selected device 130 toincrease. Similarly, in response to the user's head moving from anupward facing position to a more downward facing position, the processor1005 may cause a volume setting of the selected device 130 to decrease.As in the case of functions associated with the device 130, theapplication 710 may allow the user 102 to define additional functions804 associated with each known device (or group of known devices) and toselect the setting to control by the wearable EEG apparatus 200 inresponse to the change in position or orientation. In this way, a user102 may activate or deactivate a device 130 based on a level of focusand may adjust a setting of the device 130 based on movements of theuser's head.

FIG. 9A shows an example environment 900 for controlling a setting 806associated with a device 130. The environment 900 includes a light bulb130 controlled by a device controller 120. The device controller 120includes an activation controller 904 capable of switching the lightbulb 130 on and off and a dimmer controller 906 capable of adjusting thebrightness of the light bulb 130, e.g., by adjusting the current flowingthrough the bulb 130. Here, the setting 806 of the dimmer controller 906is a relatively low value, corresponding to a low level of brightness.Referring to FIG. 9B, the setting 806 of the dimmer controller 906 hasbeen moved to a higher value in response to the wearable EEG apparatus200 tilting upward. Processes of detecting the device and detectingmovement of the wearable EEG apparatus may be as those described above.For example, the system may calculate a brightness increase or decreaseto match (or be a function of) the amount by which the device's yaw hasincreased or decreased. As described previously, in response todetecting the brain activity, the wearable EEG apparatus 200 may alsotransmit a signal to the device controller 120, causing the devicecontroller 120 to turn the light on or off via the activation controller904. Using the wearable EEG apparatus 200 in this way, a user 102 mayturn the light 130 on or off based on a level of focus, and the user mayadjust the brightness of the light 130 based on movements of the user'shead. The wearable EEG apparatus 200 may be used to activate/deactivateand adjust settings of a variety of devices 130, including turning onand off and controlling the speed of a fan, activating/deactivatingaudio equipment and adjusting the volume, activating/deactivating aclimate control system and adjusting the temperature, etc. AlthoughFIGS. 9A and 9B show the wearable EEG apparatus 200 communicatingdirectly with the device 130 and its device controller 120, thecommunications may route through one or more intermediate devices suchas were shown in FIGS. 1, 6A, 6B and 7.

FIG. 10 illustrates example hardware that may be included in any of theelectronic components of the system, such as internal processing systemsof the wearable EEG apparatus 200. An electrical bus 1000 serves as aninformation highway interconnecting the other illustrated components ofthe hardware. Processor 1005 is a central processing device of thesystem, configured to perform calculations and logic operations requiredto execute programming instructions. As used in this document and in theclaims, the terms “processor” and “processing device” may refer to asingle processor 1005 or any number of processors in a set of processorsthat collectively perform a set of operations, such as a centralprocessing unit (CPU), a graphics processing unit (GPU), a remoteserver, or a combination of these. Read only memory (ROM), random accessmemory (RAM), flash memory, hard drives and other devices capable ofstoring electronic data constitute examples of memory devices 1025. Amemory device may include a single device or a collection of devicesacross which data and/or instructions are stored. Various embodiments ofthe invention may include a computer-readable medium containingprogramming instructions that are configured to cause one or moreprocessors to perform the functions described in the context of theprevious figures.

An optional display interface 1030 may permit information from the bus1000 to be displayed on a display device 1035 in visual, graphic oralphanumeric format, such on an in-dashboard display system of thevehicle. An audio interface and audio output (such as a speaker) alsomay be provided. Communication with external devices may occur usingvarious communication devices 1010 such as a wireless antenna, a radiofrequency identification (RFID) tag and/or short-range or near-fieldcommunication transceiver, each of which may optionally communicativelyconnect with other components of the device via one or morecommunication system. The communication device(s) 1010 may include atransmitter, transceiver, or other device that is configured to becommunicatively connected to a communications network, such as theInternet, a Wi-Fi or local area network or a cellular telephone datanetwork, or to make a direct communication connection with one or morenearby devices, such as a Bluetooth transmitter or infrared lightemitter.

The hardware may also include a user interface sensor 1045 that allowsfor receipt of data from input devices 1050 such as a keyboard orkeypad, a joystick, a touchscreen, a touch pad, a remote control, apointing device and/or microphone. Digital image frames also may bereceived from a camera 1040 that can capture video and/or still images.The system also may receive data from one or more sensors 1020 such asEEG sensors 220 and motion/position sensors 1070, such as inertialmeasurement sensors.

In this document, an “electronic device” or a “computing device” refersto a device that includes a processor and memory. Each device may haveits own processor and/or memory, or the processor and/or memory may beshared with other devices as in a virtual machine or containerarrangement. The memory will contain or receive programming instructionsthat, when executed by the processor, cause the electronic device toperform one or more operations according to the programminginstructions.

The terms “memory,” “memory device,” “computer-readable medium,” “datastore,” “data storage facility” and the like each refer to anon-transitory device on which computer-readable data, programminginstructions or both are stored. Except where specifically statedotherwise, the terms “memory,” “memory device,” “computer-readablemedium,” “data store,” “data storage facility” and the like are intendedto include single device embodiments, embodiments in which multiplememory devices together or collectively store a set of data orinstructions, as well as individual sectors within such devices. Acomputer program product is a memory device with programminginstructions stored on it.

The terms “processor” and “processing device” refer to a hardwarecomponent of an electronic device that is configured to executeprogramming instructions, such as a microprocessor or other logicalcircuit. A processor and memory may be elements of a microcontroller,custom configurable integrated circuit, programmable system-on-a-chip,or other electronic device that can be programmed to perform variousfunctions. Except where specifically stated otherwise, the singular term“processor” or “processing device” is intended to include bothsingle-processing device embodiments and embodiments in which multipleprocessing devices together or collectively perform a process.

An “imaging system” is any device or system that is capable of opticallyviewing an object and converting an interpretation of that object intoelectronic signals. One example of an imaging device is a digitalcamera.

A “machine learning model” or a “model” refers to a set of algorithmicroutines and parameters that can predict an output(s) of a real-worldprocess (e.g., identification or classification of an object) based on aset of input features, without being explicitly programmed. A structureof the software routines (e.g., number of subroutines and relationbetween them) and/or the values of the parameters can be determined in atraining process, which can use actual results of the real-world processthat is being modeled. Such systems or models are understood to benecessarily rooted in computer technology, and in fact, cannot beimplemented or even exist in the absence of computing technology. Whilemachine learning systems utilize various types of statistical analyses,machine learning systems are distinguished from statistical analyses byvirtue of the ability to learn without explicit programming and beingrooted in computer technology. A machine learning model may be trainedon a sample dataset (referred to as “training data”).

The term “bounding box” refers to a rectangular box that represents thelocation of an object. A bounding box may be represented in data by x-and y-axis coordinates [x_(max), y_(max)] that correspond to a firstcorner of the box (such as the upper right corner), along with x- andy-axis coordinates [x_(min), y_(min)] that correspond to the corner ofthe rectangle that is opposite the first corner (such as the lower leftcorner). It may be calculated as the smallest rectangle that containsall of the points of an object, optionally plus an additional space toallow for a margin of error. The points of the object may be thosedetected by one or more sensors, such as pixels of an image captured bya camera.

In this document, the term “wireless communication” refers to acommunication protocol in which at least a portion of the communicationpath between a source and destination involves transmission of a signalthrough the air and not via a physical conductor, as in that of a Wi-Finetwork, a Bluetooth connection, or communications via anothershort-range or near-field communication protocol. However, the term“wireless communication” does not necessarily require that the entirecommunication path be wireless, as part of the communication path alsomay include a physical conductors positioned before a transmitter orafter a receiver that facilitate communication across a wirelessposition of the path.

When this document uses relative terms of position such as “front” and“rear”, or “forward” and “rearward”, it is intended to cover anarrangement in which a device is worn by a human, with the human facingin the direction that is considered to be forward or the front.

While the invention has been described with specific embodiments, otheralternatives, modifications and variations will be apparent to thoseskilled in the art. Accordingly, it will be intended to include all suchalternatives, modifications, and variations within the spirit and scopeof the appended claims.

The invention claimed is:
 1. A method of training a machine-learningmodel, the method comprising: using a mobile device having a camera tocapture images of objects in an environment; receiving via a userinterface, a user selection of one of the objects; receiving a commandassociated with the selected object; in response to receiving the userselection, training the machine-learning model to recognize the selectedobject by: receiving one or more images of the selected object, andtraining the machine-learning model based on the one or more images;transmitting the trained model and the command to a wearable electronicdevice via a communication link between the mobile device and thewearable electronic device; and causing the wearable electronic deviceto: save the trained machine-learning model to a data store in thewearable electronic device, associate the command with the trainedmachine-learning model, and execute the command in response to thetrained machine-learning model recognizing the selected object in animage captured by the wearable electronic device.
 2. The method of claim1, further comprising: determining one or more regions of interest ofthe one or more images; and wherein training the machine-learning modelis based on the portion of the one or more images within the one or moreregions of interest.
 3. The method of claim 1, further comprising:receiving a reference image of the selected object; identifyingreference features related to the selected object in the referenceimage; constructing a bounding box around the object in one or more ofthe one or more images; extracting features from the portion of each ofthe images within the bounding boxes; comparing the extracted featureswith the reference features; and in response to a threshold number ofextracted features matching reference features, training themachine-learning model on the portion of the image within the boundingbox.
 4. The method of claim 3, further comprising: adjusting adimension, position, or orientation of the bounding box around theobject until the threshold number of extracted features match referencefeatures.
 5. The method of claim 1, wherein receiving the one or moreimages comprises capturing a video of the object at a plurality ofangles.
 6. The method of claim 1, wherein the command, when executed,causes the object to change from a first state to a different state. 7.The method of claim 1, wherein the machine-learning model comprises aneural network.
 8. The method of claim 1, wherein the machine-learningmodel is configured to: determine a probability that the object is inthe image; and recognize the object when the determined probabilitysatisfies a probability threshold.
 9. A system for training amachine-learning model, the method comprising: a mobile device having acamera to capture images of objects in an environment, wherein themobile device is configured to: receive, via a user interface, a userselection of one of the objects; receive a command associated with theselected object; and train the machine-learning model to recognize theselected object by: receiving one or more images of the selected object,and training the machine-learning model based on the one or more images;transmitting the trained model and the command to a wearable electronicdevice via a communication link between the mobile device and thewearable electronic device; and causing the wearable electronic deviceto: save the trained machine-learning model to a data store in thewearable electronic device, associate the command with the trainedmachine-learning model, and execute the command in response to thetrained machine-learning model recognizing the selected object in animage captured by the wearable electronic device.
 10. The system ofclaim 9, wherein the mobile device is further configured to: determineone or more regions of interest of the one or more images; and train themachine-learning model based on the portion of the one or more imageswithin the one or more regions of interest.
 11. The system of claim 9,wherein the mobile device is further configured to: receive a referenceimage of the selected object; identify known features related to theselected object in the reference image; construct a bounding box aroundthe object in one or more of the one or more images; extract featuresfrom the portion of each of the images within the bounding boxes;compare the extracted features with the known features; and in responseto a threshold number of extracted features matching known features,train the machine-learning model on the portion of the image within thebounding box.
 12. The system of claim 11, wherein the mobile device isfurther configured to: adjust a dimension, position, or orientation ofthe bounding box around the object until the threshold number ofextracted features match known features.
 13. The system of claim 9,wherein receiving the one or more images comprises capturing a video ofthe object at a plurality of angles.